[STACKED on #968] fix(vmm): bound the logs a CVM writes within a boot - #970

STACKED PR — merge #968 first.

Problem

The logs a CVM writes into its work directory are unbounded within a boot. QEMU appends to serial.log for the whole life of a boot, and the supervisor appends to stdout.log / stderr.log across boots without ever clearing them. A chatty or malicious guest can fill the host disk, and the archiving step at the next boot read the whole serial log into memory in one allocation.

This PR started from a narrower symptom — serial-log trimming discarded the boot delimiter along with older output — but that symptom only existed because several boots shared one capped archive file and the delimiter had to be recovered from the log text. Bounding a boot removes the condition, so the PR now fixes the cause and the original workaround is deleted.

Why QEMU cannot do this

The QAPI schema for the file and pty chardev backends exposes only out / in / append plus the generic logfile / logappend. None is a size limit. The one size knob in the chardev family belongs to ringbuf, an in-memory ring rather than a file. libvirt hit the same wall and answered it with a separate daemon, virtlogd.

Implementation

Rotation lives in a new logrotate module that works on a path and knows nothing about what it is rotating — the same way logrotate(8) does not care. <path>.N-1 becomes <path>.N, the live file is archived as <path>.1, and the oldest segment is discarded. A VM start is simply another rotation trigger, so the previous boot is preserved as .1 and boot boundaries land on segment boundaries.

Three details carry the design.

logappend=on is now passed to the chardev. QEMU otherwise opens the log without O_APPEND and keeps writing at its stale offset after a truncation, punching a sparse hole that leaves the file as large as it was. Measured on QEMU 8.2.2, truncating a 100000-byte log then writing 10 bytes:

This requirement is the module's documented contract. It holds equally for the supervisor's OpenOptions::append(true) sinks, which is why stdout/stderr are a call site rather than a second implementation.

The live file is truncated in place, never renamed. The writer holds an open fd on it, so a rename would leave it appending into an unlinked inode, losing every later line without an error. A test pins the inode across rotation.

Truncating to zero rather than compacting to a retained buffer is what keeps the log-viewing API working. A follower sees the file shrink and resumes at offset 0, where there is now nothing to re-read. Verified against the tail process tailf spawns: across a rotation of a 489 KB file a follower received its three trailing lines and then the three new ones, with no duplication. Compacting in place instead replayed the entire retained file to every connected viewer on every rotation.

All three logs, but not gated alike

The supervisor already opens its sinks with append(true) and reopens them when they change — its own comment calls that "logrotate detection" — so those two satisfy the contract regardless of which VMM launched the VM, and keep their cap across a VMM upgrade.

serial.log needs the gate. The supervisor is a detached daemon and owns the QEMU processes, so restarting VMM leaves running CVMs untouched; after an upgrade those processes are the ones the previous binary launched, without logappend=on. Rotating their logs would punch the sparse hole above, the cap would never hold, and every subsequent tick would rotate again. Eligibility is therefore recorded on the process itself, in the ProcessAnnotation the supervisor already stores: it describes the QEMU that is actually running, and an annotation written by an older VMM has no such key and deserializes to false. Reading the recorded argv would not work — TPM-backed VMs run through vm-launcher, whose argv is only ["vm-launcher", "--spec", <path>].

serial.history.log removed

The archive existed because the live log was unbounded and QEMU truncated it at every boot. Both premises are gone. rotate_serial_log, trim_serial_history, serial_history_max_bytes and serial_history_file are deleted, along with the splice branch that reconstructed a boot delimiter when one boot overflowed the archive cap — segments carry that structure in the filesystem instead, so the guest can no longer confuse a boundary search by printing delimiter-shaped output of its own.

Removing the config key is safe: CvmConfig does not deny unknown fields, so a vmm.toml that still sets it keeps loading. Existing serial.history.log files are left on disk; they are bounded and harmless.

Configuration

[cvm.log]
max_bytes = "4M"
max_backups = 3
check_interval_secs = 5

Named for the mechanism rather than for serial, because stdout and stderr use the same section. The values live in the shipped vmm.toml rather than in serde defaults, so the defaults are themselves parsed on every load — which is not cosmetic: the doc comment inherited from serial_history_max_bytes advertised sizes like "4MB", which size_parser does not accept. A serde default had been hiding the fact that anyone following that comment would have failed to start the VMM.

Behaviour changes to be aware of

stdout.log / stderr.log previously accumulated across boots and were never cleared; they are now rotated at each VM start. Reading them still shows the current boot, with earlier boots in .1….N, and the boot separator is still written so a segment read on its own shows where a boot began.

Worst-case disk per VM is now 3 logs × (1 + max_backups) × max_bytes = 48 MB at the defaults. QEMU's own stdout/stderr are usually tiny and rarely rotate, so the practical figure is far lower.

Known limitation

The log API serves only the live file, so rotated segments are not readable through it and a rotation leaves a reader with an empty file. One line is written into the emptied log saying where the output went, so the absence is self-explanatory:

===== rotated 4194304 bytes to serial.log.1 @ 2026-08-06T00:00:00Z =====

Making ch=serial span segments is deliberately left to a follow-up.

Rotation is triggered by a periodic stat, so a live log can overshoot the cap by one interval's worth of output. The overshoot only makes one segment larger and is reclaimed on the next tick.

Verification

• cargo test -p dstack-vmm --all-features: 105 passed.
• cargo clippy -p dstack-vmm --all-features and cargo fmt --check: clean.
• QEMU behaviours measured against QEMU 8.2.2; follower behaviour measured against the coreutils tail that tailf spawns.